Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shell script rewrite to comply with POSIX and best practices #1908

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

9ao9ai9ar
Copy link

@9ao9ai9ar 9ao9ai9ar commented Oct 17, 2024

Resolves #1855
Resolves #1446
Fixes #1810

This is not the first POSIX solution to have ever been requested or presented, though certainly the most comprehensive one yet. Regrettably, the robustness goal hasn't truly been realized, so it would need to be addressed in a separate pull request. I'd also like to point out that alternatives like the one in #1696 exist and are worth considering.

Target Platforms for Modern Desktop Versions of Firefox

Based on the supported build hosts and targets of Firefox.

Tier-1

  • Windows (WSL, Cygwin, MSYS2, etc.) - Not targeted by the shell scripts.
  • macOS - Automated testing only.
  • Fedora
  • Debian
  • Ubuntu

Tier-2

  • openSUSE
  • Arch
  • Alpine
  • NixOS - Requires the psmisc or lsof package for prefsCleaner.sh to work.

Tier-3

  • FreeBSD
  • OpenBSD
  • NetBSD
  • Solaris - Version 11 and above only.

Non-Tiered

  • DragonFly
  • OpenIndiana
  • Haiku (Iceweasel)

POSIX Compliant Shells

@MagicalDrizzle
Copy link
Contributor

prefCleaner works fine for me, however updater doesn't seem to be able to append overrides.

@MagicalDrizzle
Copy link
Contributor

Seems to be just a broken check? This fixes it and updater seems to work fine now.
if ! [ "$SKIPOVERRIDE" ]; > if [ "$SKIPOVERRIDE" = false ];

@MagicalDrizzle
Copy link
Contributor

MagicalDrizzle commented Oct 24, 2024

Error in both scripts - you were checking for the literal file named SCRIPT_FILE:

// from
SCRIPT_FILE=$(preadlink "$0") && [ -f SCRIPT_FILE ] || exit 1
// to
SCRIPT_FILE=$(preadlink "$0") && [ -f $SCRIPT_FILE ] || exit 1

and for your missing /bin/sh question - no it doesn't seem to work.

@9ao9ai9ar 9ao9ai9ar changed the title Make shell scripts more portable and POSIX-compliant Rewrite into more POSIX-compliant, portable, robust, composable and readable shell scripts Oct 24, 2024
@jamesericdavidson
Copy link

@9ao9ai9ar Thoughts on using #!/usr/bin/env? sh could be busybox, the gimped version of Bash (which is not POSIX-compliant), etc.

@9ao9ai9ar
Copy link
Author

9ao9ai9ar commented Oct 29, 2024

@9ao9ai9ar Thoughts on using #!/usr/bin/env? sh could be busybox, the gimped version of Bash (which is not POSIX-compliant), etc.

The POSIX way to do it is to overwrite the shebang with the path of sh as determined by getconf PATH at install-time (see the relevant section in the POSIX document), but I think that is overkill and I would just ask the users to change it themselves if needed. My personal opinion is that /bin/sh works better than /usr/bin/env sh for the shebang due to the following reasons:

  1. /usr might not be mounted.
  2. env might not be located in /usr/bin.
  3. On some systems, at most one argument can be passed to the shebang interpreter, which is already used up in the case of #!/usr/bin/env sh, but in the alternative, we could, for example, turn on tracing by writing #!/bin/sh -x.
  4. It is more efficient to execute /bin/sh than /usr/bin/env sh.
  5. The result of running /bin/sh is not affected by the value of PATH, unlike /usr/bin/env sh, so there are fewer surprises.
  6. The /bin/sh in Solaris 10 and earlier does not comply with POSIX as it predates the standards, but we don't have to worry about that since modern Firefox has only been made available in Solaris beginning with version 11.

@aartoni
Copy link

aartoni commented Nov 16, 2024

Hey @9ao9ai9ar @MagicalDrizzle I've noticed that progress on this PR is slowing down, so I'm willing to create a checklist of things to ensure that the script is working properly, and then make a GitHub workflow to test the script on different platforms.

Here are some things to be checked:

  • normal execution returns EX_OK
  • check every non-zero EX_*
  • user.js gets updated
  • user-overrides.js is respected

This will tick the GitHub available runners off the list of things that testers have to check manually. Manual tester can then copy the workflow to test locally.

What do you think? An alternative would be splitting this PR into smaller PRs.

@9ao9ai9ar 9ao9ai9ar requested a review from sertonix November 30, 2024 21:46
@9ao9ai9ar 9ao9ai9ar marked this pull request as ready for review November 30, 2024 21:46
@MagicalDrizzle
Copy link
Contributor

@9ao9ai9ar prefsCleaner works, updater only partly - works only if there's already an user.js file. It won't download a new user.js if doesn't exist, then will try to backup the nonexistent user.js file which fails.

+ userjs_backup=/home/demo/Downloads/profile/userjs_backups/user.js.backup.2024-12-01_0511
+ mkdir -p /home/demo/Downloads/profile/userjs_backups
+ cp /home/demo/Downloads/profile/user.js /home/demo/Downloads/profile/userjs_backups/user.js.backup.2024-12-01_0511
cp: cannot stat '/home/demo/Downloads/profile/user.js': No such file or directory
+ userjs_backup=

@9ao9ai9ar 9ao9ai9ar marked this pull request as draft December 7, 2024 07:36
@9ao9ai9ar 9ao9ai9ar marked this pull request as ready for review January 16, 2025 02:08
Copy link

@aartoni aartoni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like your improvements to the pipeline, I find it way more readable than my version. Are you planning on getting this merged now? I'd be interested in contributing a few more changes after this gets merged, such as an extra step for checking for normal execution

@9ao9ai9ar
Copy link
Author

9ao9ai9ar commented Feb 23, 2025

Are you planning on getting this merged now?

There are still, I think, three things I'd like to do before getting this merged:
  1. Split the arkenfox_updater_check_nonroot function into two to restore the old behavior of checking for root owned files only after running the self update process, so that the script can still update itself in the absence of a Firefox profile.
  2. Properly squash, redo and consolidate the commits and commit messages one more time (apologies for breaking my previous promise). I am aiming for three commits after the rebase: my shell script rewrite, @aartoni's testing workflow contributions, and my improvements to the workflow file. The rebase will do away with misleading commit messages (e.g. 9ao9ai9ar@8317f14 isn't actually runnable on less popular platforms yet). Initially, I only pushed my branch to GitHub to have a remote backup without expecting people to review or fork it, but I was clearly wrong. Thankfully, this is more a rewrite than a refactor, so it's less important to do clean and piecemeal commits, in my humble opinion.
  3. Now that I have put it behind me, reword the comments in the GitHub workflow file in a more pleasant tone?

Yes, it's ready for merging.

I'd be interested in contributing a few more changes after this gets merged, such as an extra step for checking for normal execution

You might want to check out ShellSpec. Writing the tests in the workflow file is cumbersome, limiting and a form of vendor lock-in. For example, I can't just run the tests in my VM without setting it up as a self-hosted runner on GitHub or relying on tools such as act, which you mentioned has some differing behaviors from the hosted runners. Ideally, the only workflow step you need after installing and setting up the shells is to call ShellSpec or some command to invoke the test scripts.

9ao9ai9ar and others added 2 commits February 24, 2025 20:23
The rewrite is focused on the following five areas of interest:
1. Portability. The scripts have been tested to work in recent versions
   of the following operating systems and shells: macOS, Linux (Fedora,
   Debian, Ubuntu, openSUSE, Arch, Alpine, NixOS), BSD (FreeBSD,
   OpenBSD, NetBSD, DragonFly), SunOS (Solaris, OpenIndiana), Haiku;
   bash, dash, ash, ksh, oksh, zsh, XPG4 sh, pdksh, mksh, yash, posh,
   gwsh, bosh, osh.
2. Robustness. Employ secure shell scripting techniques, incorporate
   battle-tested open source code, clear all ShellCheck warnings, and
   fail early.
3. Composability. Put (almost) everything inside functions and make the
   scripts dot source friendly.
4. Consistency. Use tput to abstract away terminal color codes, write
   templated diagnostic messages and follow conventions in the use of
   exit status and redirections.
5. Readability. Comment extensively, assign descriptive names to
   variables and functions, and use here-documents to ease reading and
   writing multi-line messages.

Known behavioral changes:
1. There are changes to the way some options are parsed and acted on.
   For example, when both the -l and -p options are specified, -l will
   be ignored; in the old behavior, the last specified option would
   take effect. Also, an old quirk where passing the argument 'list' to
   -p was equivalent to specifying the -l option has been fixed.
2. The -h, -l and -p options of updater.sh have been added to
   prefsCleaner.sh as well.
3. All temporary files are now created using mktemp and no longer
   actively deleted, so users won't find them in the working directory
   anymore in the case of error.
4. The old prefs.js cleaning logic, which relied on non-POSIX features,
   is not preserved in the rewrite.

Resolves arkenfox#1855
Resolves arkenfox#1446
Fixes arkenfox#1810
@9ao9ai9ar 9ao9ai9ar changed the title Rewrite into more POSIX-compliant, portable, robust, composable and readable shell scripts Shell script rewrite to comply with POSIX and best practices Feb 24, 2025
@aartoni
Copy link

aartoni commented Feb 27, 2025

Thanks for the pointers!

@9ao9ai9ar 9ao9ai9ar mentioned this pull request Mar 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Make Updater.sh shell agnostic updater.sh returns with exit code 0 when failing in update_userjs
5 participants